The research in the field of human-machine interaction (“HMI”) has been expanding in these last decades. Several modern breakthroughs such as tele-operated robotics, immersive virtual reality or augmented reality have pushed the limits and capacity of humans to interact with their environment a step further. Visual and auditory feedbacks are reaching such levels of realism that users can believe the environment they see and hear, be it remote or virtual, is physically tangible.
The growing variety of tasks that can be performed by controlling a teleoperated manipulator, such as robots or less automated devices, and their increasing importance call for more intuitive and reliable human-machine interfaces. Besides the evident advantage of allowing the user to control a manipulator from a remote location, teleoperation can also be used to enhance the user's perception by increasing the strength of a stimulus, be it visual or tactile, to filter the user's tremor, scale down/up his movements as well as the force applied, thus allowing for operation in restricted or unfriendly environments. Nowadays, teleoperated robots are used to inspect nuclear reactors, for rescue in catastrophic situations, for marine applications, or to perform minimally invasive surgeries (“MIS”), one of the most promising research directions where errors in the bidirectional communication with the user cannot be tolerated.
Also the use of robotic systems to perform complex surgical procedures is steadily increasing. While these robotic systems can improve performance, they currently provide visual feedback alone thus depriving the surgeon of important haptic information. Haptic feedback can be divided into force and tactile feedback, both essential for dexterous manipulation. Without force feedback the surgeon using a position-controlled teleoperated robot has a serious risk to damage internal tissues by applying too much force. In addition, some works have emphasized the benefits of introducing force feedback in robotic surgical systems in terms of usability and reduction of cognitive load. Our multimodal tactile perception allows us to discriminate the objects we manipulate based on their texture, compliance, shape and thermal characteristics by integrating these sensorial inputs in a congruent percept. By palpating the tissues, the surgeon can find hidden arteries based on pressure cues, for example the pulse, and discern cancerous tissues based on temperature alone. Interestingly, even in the absence of other relevant tactile cues, we are able to discriminate a variety of materials based on thermal cues alone. Another important factor for fine manipulation of a teleoperated system is its embodiment. Force feedback has been shown to increase embodiment of a virtual tool. Similarly, embodiment being a multisensory process, it is likely dependent on the congruency of the multimodal feedback as well.
Several haptic devices combine tactile and kinesthetic feedback, often by mounting a tactile display on a force feedback device. Recently, displays presenting several tactile modalities simultaneously to the user were designed. While current designs have reached satisfactory tactile stimulation, they are still limited due to their size and rigidity. Moreover, the haptic feedback, for example the combination of tactile and kinesthetic feedback, which is needed to truly make the environment tangible is of absent or considerably lacking realism compared to its visual and auditory counterparts. Most haptics actuators and devices can only display a limited amount of haptic cues. Typical actuators used in the field are vibrating eccentric motors or similar piezoelectric actuators providing warnings rather than realistic tactile cues. In addition, human tactile perception is multimodal; we need several cues, such as compliance, texture or temperature to identify objects. While multimodal tactile actuators exist, the technology is bulky, complicated and rigid. These limitations restrict their usage to specific predefined areas of the body and limiting their wearability and their integration with other devices such as the master of a teleoperated system.
Recently, a publication successfully used thermal feedback as a vector of information in environments in which audio or vibrotactile feedback might be masked by noise or movements. Wilson et al. “Thermal Feedback Identification in a Mobile Environment,” Proceedings of Haptic and Audio Interaction Design (“HAID”), 2013. In their studies, thermal feedback is used instead of vibration to provide warnings about incoming cell phone messages. By varying thermal parameters such as subjective intensity, for example moderate warm, intensive cold etc., or direction of the temperature change, it was possible to inform the user on the urgency of the message. For example, very warm was used for urgent, mild warm was used for not urgent, or the identity of the sender, warm for family, cold for work for example. In addition, they found promising results for the identification of individual thermal parameters while sitting and walking outdoors. By combining thermal stimulation with multimodal tactile stimulation, a vast array of stimulations is available. These stimulations can provide rich encoded information to the user.
The importance of thermal cues in object, discrimination and their potential for telemanipulation has led to the development of various thermal displays. These devices are for the most composed of a Peltier element, a heatsink and a temperature sensor. One surface of the Peltier element is kept at a constant temperature by the heatsink while the other surface temperature is controlled using the sensor input in a feedback loop. These systems can be divided in two categories: in one the user's finger, or other body parts, is in constant contact with the display, in the other the contact between the user and the display happens only when a contact in the virtual or teleoperated environment occurs. A flexible and wearable device should therefore be in constant contact with the user.
The International patent publication No. WO 2009/061572 discloses a thermal haptic feedback system for generating thermal haptic effects. The system includes multiple cells, each of them providing heating or cooling in a generally isolated area that is approximately the size of the cell. Each cell can be independently controlled, allowing for the generation of complex thermal haptic patterns for generating haptic effects. In addition to thermal haptic effects, it is also described that each cell can include force feedback type haptic effects generated by, for example, actuators. Therefore, tactile cues are provided to the user with the use of further devices coupled to the thermal haptic system, thus rendering the system bulky and not completely compliant.
Despite some of the above-discussed advancements in the field of haptics, there is still an urgent need in the field for small, wearable, easily embeddable and multimodal tactile feedback devices.